Multi-party conversation analyzer and logger

ABSTRACT

In one aspect, the present invention facilitates the investigation of networks of criminals, by gathering associations between phone numbers, the names of persons reached at those phone numbers, and voice print data. In another aspect the invention automatically detects phone calls from a prison where the voiceprint of the person called matches the voiceprint of a past inmate. In another aspect the invention detects identity scams in prisons, by monitoring for known voice characteristics of likely imposters on phone calls made by prisoners. In another aspect, the invention automatically does speech-to-text conversion of phone numbers spoken within a predetermined time of detecting data indicative of a three-way call event while monitoring a phone call from a prison inmate. In another aspect, the invention automatically thwarts attempts of prison inmates to use re-dialing services. In another aspect, the invention automatically tags audio data retrieved from a database, by steganographically encoding into the audio data the identity of the official retrieving the audio data.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This patent application is related to U.S. patent application Ser. Nos.11/475,541 and 12/284,450, and provisional patent application61/380,325, all of which are herein incorporated by reference.

FIELD OF THE INVENTION

The field of the invention relates to prison phone systems, automaticmonitoring of phone calls and conferences, speaker recognition, speakeridentification, detection of three-way calls, audio data mining, voicelogging, non-vocabulary-based analysis of dialog, affective computing,stenographic encoding of data within audio, and more specifically toautomated analysis of multi-party conversations, and automated keywordsearching.

BACKGROUND OF THE INVENTION

As the hardware and software to record conversations in digital form hasbecome more and more affordable over recent years, recording andarchiving of conversations such as customer service calls, teleclasses,business teleconferences, and calls made by prison inmates has becomeroutine. As digital voice recorders have become more economical andeasier to use, their use for dictation and note taking has been steadilyincreasing. It is expected that with the increasing availability ofportable digital devices capable of audio recording (such as MP3player/recorders, cell phones, and digital voice recorders) willcontinue to increase for many years to come, and that the uses of thesedevices will continue to expand. Indeed, we are approaching the timewhen audio recording of one's entire lifetime experience will bepractical and economical. As the amount of monitored and stored audiodata increases, there is an ever increasing need for technological toolswhich can extract information from digital audio data. Backgroundinformation on a number of conversation-recording market segments, aswell as background information on speech recognition, voiceverification, and voice identification is presented below.

Prison Market

Modern correctional institutions face many challenges concerning phonecalls made by inmates. One intent of correctional institutions is torestrict inmates from making phone calls to persons they should not becontacting. To help accomplish this aim, many modern prison phonesystems require inmates to use an identifying PIN to use the phonesystem, and the phone system enforces limitations on numbers which canbe directly dialed, based on the prisoner's individual PIN. Many prisonphone systems, for instance, limit a given prisoner to dialing one of apre-approved set of phone numbers when making calls. One example of thetype of phone call inmates are not allowed to make is phone calls toother convicted criminals (either in other prisons, or out on parole).Another example of the type of phone call inmates are not allowed tomake is phone calls which threaten, intimidate, or harass someone.Another type of phone call prisoners have occasionally made which haveembarrassed officials is a phone call to a public radio station, wherethe prisoner winds up speaking on the radio without authorization. Oneway in which inmates circumvent the limited-dialing restrictions ofmodern prison phone systems is to make a phone call to an “allowedparty”, who has invited over to his or her residence a “disallowedparty”, who then participates in the phone call after the phone isinitially answered by the allowed party. Another way in which inmatescircumvent the limited-dialing restrictions of modern prison phonesystems is to make a phone call to a friend who has three-way calling,and then have that person conference in a person at another telephonenumber (which is not approved for the prisoner to be dialing). Anotherway that inmates circumvent the dialing limitations is to have someoneat an allowed number set their phone to call-forward to another number(which is not approved for the prisoner to be dialing). One brand ofprison phone systems boast a “third-party-call-indicating clickdetector” feature, which is designed to detect a set of supposedlytelltale click sounds on the line when a third party is conferenced onto the line. Such detectors are unfortunately unreliable at best,because many modern telephone systems don't create any particular noiseson the line when conferencing in a third party, or when forwarding acall to another number, but none the less, prison officials have beenmotivated by the promise of such systems enough to purchase phone systemupgrades. Indeed, word of the existence of such systems has spread amonginmates, along with the belief and story that if inmates in aconversation (where a third party is being conferenced in) are makingenough noise at the time of the conferencing clicks, then the systemwill not detect the clicks.

To continue to market “conference call click detecting” systems toprisons in the face of such stories, manufacturers of such systems haveutilized phone system hardware that separates the electronic signalsproduced by the phone the inmate is talking on at the prison, from theelectronic signals coming in from the outside phone network.Telecommunications with the incoming and outgoing signals separated issometimes referred to as four-wire telephony (in contrast to thetwo-wire telephony systems typically used in homes, where incoming andoutgoing signals share the same pair of wires). We will also refer tothis four-wire technique in this document as “telephonic directionalseparation”. When click detection algorithms are run on only the signalscoming in from the outside phone network, clicks can be detected (ifthey exist) regardless of how much noise a prisoner makes on a phone atthe prison. In addition to click detection methods, tone detectionmethods such as those described in U.S. Pat. No. 5,926,533 (which isherein incorporated by reference) are known in the art. However, if agiven outside phone system accomplishes call conferencing withoutcreating clicks or tones, the call conferencing can not be detectedthrough click or tone detection. There is a need for innovativetechnology which can detect conference calls in situations where notell-tale clicks or tones are present.

A compounding problem facing corrections facilities today is thatdetecting and automatically disconnecting a call based on the fact thatit is a conference call or a forwarded call may not be the right thingto do in some circumstances. For instance, if someone an inmate isallowed to call at home sets his home phone to forward to his cell phoneif he doesn't answer at home, the call should be allowed to go through.Likewise, if one person an inmate is allowed to call wants to conferencein another person that the inmate is allowed to call, such a call shouldnot be automatically disconnected. There is a need for innovativetechnology which will not interrupt conference calls and forwarded callswhich should be allowed to take place, while automatically disconnectinginstances of call forwarding and conference calling which should not beallowed to take place.

Another problem facing the management of corrections facilities today isthat a service has become available which allows automatedvoice-prompted call forwarding, including spoofing a false caller IDnumber. This service (if dialed by an inmate directly of reached by aninmate through call forwarding or a three-way call) allows the inmate toverbally speak a number to be dialed, and verbally speak a caller IDnumber to be spoofed. The service then patches the inmate through to thenumber to be dialed, and the caller ID number seen by the person at thatnumber is the spoof number verbally specified by the inmate. Suchsystems will be referred to in this document as verbally programmedre-dialers. There s a need for innovative technologies to combat the useof verbally programmed re-dialers by inmates.

Another potential problem which is of concern to some institutions isthe potential for employees of the institution to release tounauthorized persons copies of recorded phone calls. This may be ofconcern, for instance, regarding phone calls placed by “celebrity”inmates. There is a need for innovative technologies which will helpcorrectional institutions combat unauthorized release of call recordingsto unauthorized persons.

The quantity of phone calls made on a daily basis from moderncorrectional institutions is large, and even though many correctionalinstitutions record all phone calls made by inmates, it is a financiallyinfeasible task to manually review, spot monitor, or manually spotreview all phone calls made, and even if such manual monitoring werefeasible, persons monitoring the calls would be unlikely to know if agiven call was forwarded to someone at a different number than thenumber that was dialed, and the entire call might have to be listened toin order to detect an instance of conferencing in a third party. Thereis a need for more automated monitoring with innovative features whichwould statistically allow a high degree of accuracy in pinpointing phonecalls which went to an un-allowed party.

Even when inmates are talking to allowed parties, it is desirable toprevent inmates from facilitating illegal activity via their phonecalls. Techniques (such as described in U.S. Pat. No. 6,064,963, whichis herein incorporated by reference) are known in the art forautomatically spotting key words in conversations. Unfortunately it canbe difficult to know what key words to look for, because inmates knowthat all their calls are being recorded, so they may be unlikely tospeak about prohibited subjects in a directly obvious manner. Even ifprison officials reviewed all of every phone call made, it would bechallenging to figure out the meaning of what was being said if part orall of the conversation were essentially in code. There is a need fortechnological advances which can aid prison officials in detecting phonecalls about prohibited subjects, and there is a need for technologicaladvances which can provide prison officials with clues to help decipherconversations which are partly “in code”.

Correctional institutions are not only responsible for preventinginmates from engaging in illegal and/or harmful activities, they arealso charged with rehabilitating inmates. One key factor which can aidin rehabilitating inmates is monitoring each prisoner's psychologicalstate of mind. Monitoring inmates' phone calls can give excellent cluesto inmates' states of mind, but prisons don't have the budget to haveeven unskilled personnel monitor the majority of phone calls made, andthe level of training and attentiveness that would be required tomonitor the majority of phone calls and keep psychological notes is notreasonably feasible for prisons to expend. There is a need forinnovative technology and automated systems to help prison officialstrack the psychological states of mind of inmates.

Another challenge facing prison officials is the challenge ofmaintaining certainty about who is making which calls. Although manymodern prison phone systems require a prisoner to enter a PIN to makecalls, it is still possible for inmates to share PINs with each other,which gives them access to dialing numbers which are not on their“allowed phone number” list. There is a need for more reliable ways forprison officials to be able to detect when inmates are directly dialingnon-allowed phone numbers by using identifying information of otherinmates. It has been proposed to use digital signal processing SpeakerIdentification techniques (such as those described in U.S. Pat. No.6,519,561, which is herein incorporated by reference)) in place of PINsto identify which inmate is making a call, but speaker identificationtechnology is nowhere near as reliable as fingerprinting, so such anidentification system has not been deemed a viable substitute for PINs.

Speaker recognition technology relies on extracting from human speechcertain characteristics of the fundamental vibration rate of thespeaker's vocal chords, and certain information about the resonances ofvarious parts of the vocal tract of the person speaking, which areindicative of the physiology of that particular person's vocal tract.There are two problems that lead voiceprints to be far less individuatedthan fingerprints. The first problem is that there is not as muchvariation in the physiology of typical people's vocal tracts to provideas rich a differentiation as fingerprints provide. The second problem isthat each given person's vocal tract characteristics actually vary in anumber of ways depending on time of day, how much the person has beentalking that day and how loud, whether or not the person has a cold,etc.

Some modern prison phone systems use voice verification in conjunctionwith PINs, to make it more difficult for one inmate to falsely identifyhimself as another inmate. Voice verification has less stringentrequirements than voice identification. In voice verification, thesystem is typically simply ensuring that the person speaking a passphrase has a voice that is “close enough” to the voice characteristicsof the person who's PIN is used. Even with voice verification augmentingPIN usage, one inmate might “share his identity” with another inmate, byentering his PIN and speaking his pass phrase, and then handing thephone off to another inmate. Or an inmate may use a pocket dictationrecorder to record another inmate's pass phrase, and then play it intothe phone at the appropriate time. There is a need for more robustinmate identification technology which prevents one inmate from “handingoff” his identity to another inmate in a way that would allow the otherinmate to make calls to numbers which he would otherwise not be allowedto call.

The only data most modern prison phone systems keep track of and makeeasily searchable are records of numbers dialed, and time, date, andduration of calls, inmate who originated the call, reason for calltermination (regular termination, 3-way call termination, out-of-money,etc.), type of call (collect, prepaid, debit, etc.). There is a need fortracking innovative metrics which allow prison officials to moreaccurately pinpoint which call recordings are worthy of human review,and speech-to-text technologies only partially address this need. It mayfor instance be desirable to detect when an inmate is giving orders orthreatening someone. This may be difficult to do from vocabulary alone,especially since the prisoner knows the call is being monitored, and maytherefore speak “in code”. There is also a need for innovativetechnologies which offer real-time detection of prohibited calls(through detection of non-allowed call participants, and/or through thenature of the dialog between the inmate and the called party orparties), and there is the need for a system which offers prisonofficials the opportunity to quickly make a decision in real time as towhether a given call should be interrupted, and interrupt the phone callif needed based on real-time content of the call.

The quantity of phone calls made in prisons on a daily basis isenormous. Even the quantity of such phone calls which might containconversations which merit attention by investigators is substantial.There is a need for innovative technologies which can present in aneasily graspable form a summary of calls which might meritinvestigation. There is a further need for innovative technologies whichcan sort calls by criteria relevant to why the calls might meritinvestigation, and “how strongly” such calls might merit investigation.

Customer Service Market

In the customer service industry, it is common for all calls to berecorded, and for a cross-section of calls to be monitored live andother calls to be reviewed later with the aim of furthering the trainingof customer service representatives, and increasing customer retention.The increased use of Interactive Voice Response (IVR) systems in themodern customer service industry has in many cases exacerbated thefrustration that consumers experience, because one is oftencommunicating with a computer rather than a person when initiallycalling a customer service department. Some companies have recently madeavailable software designed to detect frustration on the part ofconsumers dealing with customer service departments. There is a furtherneed for innovative technologies which can aid in real-time detection oflive conversations (between customers and customer service agents) thatare “not going well”, so that customer service agents have theirsituational awareness increased, and/or customer service supervisors canintervene, possibly saving valuable customer loyalty. There is also aneed for innovative technologies which can give customer service agentsfeedback and coaching to help them deal more effectively with customers.

As in other industries where large numbers of phone calls are monitored,today's technology makes it easy for a company to record and archive allcustomer service phone calls, but technologies are lacking in the areaof automatically sorting recorded calls and flagging which ones are goodcandidates to be listened to by persons aiming to glean criticalinformation, or insights which could be used to improve customerservice. One challenge facing customer service call center managers isfinding a way to usefully keep easily searchable records containingrelevant data about recorded phone calls. Companies such as CallMiner,Inc. have begun to make products and services available which useLarge-Vocabulary Continuous Speech Recognition (LVCSR) speech-to-textconversion to convert archived audio to text. While today'slarge-vocabulary continuous speech recognition technology has achievedreasonable accuracy when trained for a particular user, it is far lessaccurate in converting speech for users for who's speech the system isnot trained, and further accuracy problems crop up when convertingspeech of more than one person in a multi-party conversation. Never theless, products and services such as those offered by CallMiner, Inc.have reached the point where their phrase and word searching functionshave been deemed useful by many customer service groups.

In some customer service departments, recording of customer servicephone calls also serves the purpose of legal documentation. This istrue, for instance, in financial institutions such as banks andbrokerages.

Speech Processing

Computational techniques of converting spoken words to text or phonemes(speech recognition), and techniques for identifying a person by voiceautomatically (speaker identification) and techniques for automaticallyverifying that particular person is speaking (speaker verification)typically employ techniques such as spectrographic analysis to extractkey features of different people's voices. The following two paragraphsare included to familiarize the unfamiliar reader with some terms andgraphical representations used in spectrographic analysis.

A black & white spectrogram of the utterance “phonetician” (thetime-domain waveform of which is shown in FIG. 2) is shown in FIG. 3.The spectrogram may be thought of as being composed of a set of verticalstripes of varying lightness/darkness. Each vertical stripe may bethought of as representative of the frequency vs. amplitude spectrumresulting from a Fourier transform of a short time window of thetime-domain waveform used to derive the spectrogram. For instance, thespectrum of a short time slice starting 0.15 seconds into the utterancewho's spectrogram is depicted in FIG. 3 (representing the spectrum ofthe beginning of the “o” vowel in “phonetician”) may be representedeither by the graph in FIG. 4 or by the vertical stripe 300 of thespectrogram in FIG. 3. The dark bands of vertical stripe 300 may bethought of as representing the peaks of the spectrum in FIG. 4. Thus aspectrogram represents a series of spectral snapshots across a span oftime. An alternative way of representing a spectrogram is shown in FIG.6, where the sequential time slices are assembled in a perspective viewto appear as a three-dimensional landscape.

The peaks in the spectrum in FIG. 4 (or equivalently, the dark bands instripe 300) are referred to as the formants of speech. These peaks fallon harmonics of the fundamental vibration rate of the vocal chords asthe speaker pronounces different utterances, and their relative heightsand how those relative heights change throughout speech are indicativeof the physiology of the particular speaker's vocal tract. Both thefundamental vibration rate of the vocal chords (shown in FIG. 3 over thetime span of the utterance of FIGS. 2 and 6) and the relative amplitudesof the speech formants vary over time as any given speaker speaks.Speaker recognition and speaker verification utilize the differencesbetween the spectral characteristics (including variations over time anddifferent utterances) of different peoples voices to determine thelikelihood that a particular person is speaking Various techniques areknown in the art for extracting from a speech sample spectral data whichmay be viewed as indicative of identifying characteristics of thespeaking person's vocal tract. Such data is commonly referred to as avoice print or voice signature. The fundamental vibration rate of agiven person's vocal chords (and certain other geometric characteristicsof that person's vocal tract) can and often does vary with time of day,length of time the person has been continuously talking, state ofhealth, etc. Thus voiceprints are not as invariant as finger prints.

Speech recognition technologies for use in such applications asspeech—to-text conversion have been commercially available in productssuch as Dragon Naturally Speaking™ (made by Nuance Communications Inc.)and ViaVoice™ (made by IBM) for a number of years now, and recentlyresearchers have also begun to develop software for recognizing theemotional content of speech. The word prosody (defined at PrincetonUniversity as “the patterns of stress and intonation in a language”) isoften used in the field of affective computing (computing relating toemotion) to refer to emotion-indicating characteristics of speech.Prosody measurements may include detecting such speech characteristicsas word rate within speech, perceived loudness, sadness, happiness,formality, excitement, calm, etc. Perceived loudness is distinguishedhere from absolute loudness by the way the character of someone's voicechanges when he or she yells as opposed to talking normally. Even ifsomeone used a “yelling voice” quietly, one would be able to understandthat the voice had the character of “yelling”. Within this document, wewill expand the meaning of the word prosody to include allnon-vocabulary-based content of speech, including all emotional tonalindications within speech, all timing characteristics of speech (bothwithin a given person's speech, and timing between one person in aconversations stopping speaking and another person in the conversationspeaking), laughter, crying, accentuated inhalations and exhalations,and speaking methods such as singing and whispering. References in whichthe reader may learn more about the state of the art in prosodydetection include:

-   -   1) MIT Media Lab Technical Report No. 585, January 2005, which        appeared in Intelligent user Interfaces (IUI 05), 2005, San        Diego, Calif., USA.    -   2) R. Cowie, D. Cowie, N. Tsapatsoulis, G. Votsis, S.        Kollias, W. Fellenz, and J. G. Taylor. Emotion recognition in        human computer interaction. IEEE, Signal Processing Magazine,        2001.    -   3) P. J. Durston, M. Farell, D. Attwater, J. Allen, H.-K. J.        Kuo, M. Afify, E. Fosler-Lussier, and L. C.-H. Oasis natural        language call steering trial. In Proceedings Eurospeech, pages        1323-1326, Aalborg, Denmark., 2001.    -   4) R. Fernandez. A Computational Model for the Automatic        Recognition of Affect In Speech. PhD thesis, MIT Media Lab,        2004.    -   5) H. Quast. Absolute perceived loudness of speech. Joint        Symposium on Neural Computation, 2000.    -   6) M. Ringel and J. Hirschberg. Automated message        prioritization: Making voicemail retrieval more efficient. CHI,        2002.    -   7) S. Whittaker, J. Hirschberg, and C. Nakatani. All talk and        all action: Strategies for managing voicemail messages. CHI,        1998.        The above references are herein incorporated by reference.

Within this document, the terms “voice print”, “voice signature”, “voiceprint data”, and “voice signature data” may all be used interchangeablyto refer to data derived from processing speech of a given person, wherethe derived data may be considered indicative of characteristics of thevocal tract of the person speaking. The terms “speaker identification”and “voice identification” may be used interchangeably in this documentto refer to the process of identifying which person out of a number ofpeople a particular speech segment comes from. The terms “voiceverification” and “speaker verification” are used interchangeably inthis document to refer to the process of processing a speech segment anddetermining the likelihood that that speech segment was spoken by aparticular person. The terms “voice recognition” and “speakerrecognition” may be used interchangeably within this document to referto either voice identification or voice verification.

In order for the voices of a given person to be identified or verifiedin voice identification processes, a sample of that person's speech mustbe used to create reference data. This process is commonly referred toas enrollment, and the first time a person provides a speech sample iscommonly referred to as that person enrolling in the system.

There are several ways that voice recognition algorithms can be thoughtof as testing a given person's voice to see if it matches a previouslystored voice print. The first way is that the voice print data can bethought of as a numerical vector derived from the reference speaker'svoice. A second numerical vector can be derived in a like manner fromthe voice under test, and a numerical algorithm can be used to comparethe two vectors in a way where the comparison produces a single numberthat has been found to be indicative of the likelihood of a correctmatch.

Since the absolute likelihood of a correct match is not independent ofthe voices of all the people who might be tested who are not a match, amore useful method compares the voice signature of the person beingtested to voice signatures from a number of other individuals, or to anaverage voice signature derived from a number of people. The likelihoodthat the voice signature under test is the voice that was used to derivethe reference voice signature is then derived from the extent to whichthe voice signature under test matches the reference voice signaturebetter than it matches other individual voice signatures, or the extentto which the voice signature under test matches the reference voicesignature better than it matches the “average” voice signature of apopulation.

A third way that voice recognition algorithms can be thought of astesting a given person's voice to see if it matches a previously storedvoice print is that the stored voice print may be thought of as a modelwhich is repeatedly tested against over time using small samples of thevoice under test, and the resulting test scores are averaged over time.This procedure may be used with one of the above methods to produce alikelihood score which has more certainty the longer the speech undertest is listened to. This variable sample length method may haveadvantages in live monitoring applications and in applications where itis desirable not to waste computational resources once a desiredcertainty level has been attained.

SUMMARY OF THE INVENTION

It is an object of the present invention to facilitate the investigationof networks of criminals, by gathering associations between phonenumbers and voice print data and names of persons reached at those phonenumbers.

It is a further object of the present invention to provide innovativetechnologies to help prisons combat the use of verbally programmedre-dialers by inmates.

In one aspect, the present invention requires both inmates and calledparties to state the name of the called party before being connected,and both stated names are converted to text. Spoken names, voice modeldata from spoken names, and text-converted spoken names are saved in thecall database of the institution recording the call. In a preferredembodiment, phone numbers dialed, spoken names, voice model data fromspoken names, and text-converted spoken names are saved in a centralizeddatabase (outside of any particular institution) as well, to provide theability for one institution to access phone number/voice data/namepairings recorded by other institutions, to aid in investigations andthe uncovering of criminal associations.

In another aspect, the present invention improves on the practice(common in many prisons) of allowing visitation phone calls withoutidentifying visitors in the phone call. In one aspect, the presentinvention has inmates identify themselves by PIN and/or spoken name, andidentify visitors by name when dialing. In another aspect, the presentinvention has visitors identify themselves by name before beingconnected to the inmate they wish to talk to. In a preferred embodiment,both the prisoner's enunciation of the visitors name and the visitor'senunciation of his or her own name are recorded and converted to speech.Additionally, brief voice model data is derived from the visitor'senunciation of his or her name, and more precise voice model data isderived from subsequent speech of the visitor during the phone call. Ina preferred embodiment, these derived data are stored in theinstitution's call database, building over time a set of voice models,recorded enunciated names, and test-converted names associated withpersons with whom a particular inmate converses. In a preferredembodiment, photographic images taken of visitors during visitationconversations are also stored in the institution's call database.

In another aspect, the present invention allows investigators to listento phone call recordings using headphones or ear-buds where the callingparty side of the line is played in one channel (for instance the leftchannel) and the called-party side of the line is played in the otherchannel (for instance the right channel). This allows intuitivediscernment of whether a new voice is on the called-party side of theline or the calling-party side of the line.

In an embodiment of the present invention for use in monitoringconference calls live or reviewing recorded conference calls, automaticvoice recognition of who is speaking enables graphical display of theidentity of the current speaking party (by photo icon or avatar or bytext name identifier, or by verbally enunciated identity in response toa key press, mouse click, or DTMF query.

In a preferred embodiment of the present invention, digital signalprocessing to carry out voice model comparison and other voice analyticfunctions is performed on a Graphics Processing Unit (GPU) card such asGPU cards used in computer gaming.

In a preferred embodiment, detection of a likely 3-way call eventtriggers an automated search for the identity of the 3-way-called party,and possibly automatically searches for past and/or future occurrencesof a newly detected voice.

In a preferred embodiment, when the present invention detects anidentity scam (PIN Sharing event) it feeds the recorded incident intoits suspicious calling pattern detection system as one of a multitude offactors used by the system for uncovering covertly made potentiallycriminal calls.

In a preferred embodiment, the suspicious calling detection system addsto the calls suspicious score for each suspicious contributing event.Different types of events are given different weights, and the highinterest class of the inmate caller may also change the weighting of anevent and the total suspicious score. In a preferred embodiment, auser's work queue can be sorted by highest suspicious scoring calls, bycase, or a number of other different factors. Highly Suspicious callsare automatically presented to users for investigation. If a userconfirms the call as suspicious after reviewing it, the system uses theconfirmed pattern to identify other historical and future similarpatterns and provides an appropriate weighting to such call patterns.

In a preferred embodiment, events which contribute toward thesuspiciousness scoring of a given call include:

-   -   identity scamming detection    -   3-way calls    -   spoken phone number detected    -   added called party    -   high interest group inmate    -   high interest group called party    -   newly identified link    -   matches historical prior confirmed suspicious calling pattern    -   new called party voice seen on call.    -   previously identified high interest called party voice detected    -   whispering detected

In a preferred embodiment, when an individual who has been an inmate ata facility (and who's voice has been enrolled in the system) leaves thefacility in which he or she has been incarcerated, the systemautomatically begins to search for the voice of that individual on phonecalls or all inmates with whom that individual had contact in thefacility as an inmate, and the present invention can be configured totake automated action (such as automatically cutting off a call), or toflag the recordings of calls where inmate-to-prior-inmate conversationsare detected.

In a preferred embodiment of the present invention for use incorrectional institutions, rather than performing continuous or periodicvoice verification during a call to detect if the voice of the inmate onthe call ever becomes an unlikely match to the voice model of the inmatewho placed the call, the system instead performs continuous periodicimposter monitoring, where the voice on the inmate side of the line isperiodically checked against voice prints or voice models of persons whoit is deemed could possibly get on the line in place of the inmate whoplaced the call. In a preferred embodiment, voice models of persons whohave been known in the past to have perpetrated identity theft fromother inmates will be preferentially monitored for. In a preferredembodiment, when an imposter is detected, the present invention may beset to perform an automated action (such as flagging the call in aparticular way or cutting off the call in real time), depending on theidentity of the imposter detected.

In a preferred embodiment, when conditions indicative of a 3-way callare detected when monitoring a call live or reviewing a recorded call,speech-to-text conversion is used to convert any spoken phone numberand/or name detected on the line in the temporal vicinity (for instancewithin 30 seconds) of the detected 3-way call event. In a preferredembodiment, a voice model is made of any new person to join theconversation during the detected likely 3-way call event, andstatistical pairings are built up over time in both the call database ofthe institution, and in a centralized database whish spans manyinstitutions. Thus over time the present invention enables lawenforcement officials to build a database of the names and phone numbersof individuals who inmates illicitly call through three-way call events,call forwarding, and the like. If on one call the phone number and voiceare both detected, the centralized database now “knows” that pairing. Ifon another call the name and voice are paired, the two events areautomatically correlated and now the phone number, voice model, and nameof the illicitly called person are all known, and in the future,detecting of the voice or number or name will suffice. In this document,methods for dealing with three-way calls and illicitly forwarded callswill be deemed interchangeable in most cases.

In a preferred embodiment, automated dialing means or SS7 telephonenetwork query means may be used to verify the “busyness” of a phonesupposed to be involved in an illicit (3-way or call-forwarded) calldetected to be likely in real time. In addition, automated orhuman-assisted dialing of a the previously verbally detected phonenumber of an illicit call participant may be used to get a verifyingsample of the voice or voices of persons who can be reached at thatphone number.

A preferred embodiment of the present invention allows officials usingthe present invention to create 3-way voice searches (searches for othercalls on which voices detected during 3-way call events are detected). Apreferred embodiment also allows users to search and label called partyvoices with phone numbers and names so that the rest of the system'scall intelligence can be brought to bear on the 3-way call. A preferredembodiment of the present invention can be set to automatically searchforward and/or backward in time for other occurrences of a voicedetected during a likely 3-way call event, wither only in other calls ofthe inmate involved, or across calls of a number of inmates, to discoverif some 3-way-called numbers and/or persons are in common to a-way callevents detected on multiple inmates' calls.

A preferred embodiment of the present invention employs directionalmicrophones or an array of microphones which may be post-processed orreal-time-processed to effectively act as one or more directionalmicrophones, to monitor conversations of inmates in the vicinity ofinmate phones while other inmates are engaged in conversations on thosephones. In a preferred embodiment, the present invention mayadditionally or alternately use electronic cameras to monitor inmates inthe vicinity of inmate phones. In a preferred embodiment, directionalaudio monitoring of inmates in the vicinity of phones is used toautomatically detect instances where one inmate (who is on the phone) isacting as an intermediary for another inmate who is prohibited fromtalking to the person who the inmate on the phone is allowed to talk to(and is talking to). In such a case, the voice of the disallowed inmateis recorded through a physically directional microphone (orelectronically directional array of microphones), and stored in a mannerlinked with the recording of the phone call that took place at the sametime. In a preferred embodiment, the identity of inmates in the vicinityof the inmate who is on the phone may be automatically derived fromvoice parameters or photographically, or may later be manually derivedphotographically.

In another aspect, the present invention automatically detects theinteractive voice response cues of verbally programmed re-dialers, andmay be configured to foil inmate attempts to use such systems. In oneconfiguration, the present invention may be set up to automaticallyterminate calls on which the automated IVR voice of such systems isdetected. In a preferred embodiment, such automatic termination takesplace after the inmate has already spoken the number to be dialed (andpossibly the caller ID to be spoofed), and before the call actually goesthrough. In a preferred embodiment, call termination can be programmedto take place in a way that the inmate will likely interpret as adropped call or an IVR interface failure, rather than an intentionallyterminated call. Thus the present invention can be programmed to “spoofcall failure” until officials decide whether it would be more useful tolet a call to the specified number go through (for instance, to furtheran investigation). A preferred embodiment of the present invention flagsall inmate calls during which automated IVR voices are detected, andautomatically converts to text numbers spoken my inmates and numbersspoken back to inmates by IVR systems. In a preferred embodiment, acentralized database of such detected numbers is maintained, and theidentities of persons at those numbers and the identities of persons whohave been dialing those numbers is maintained in a correlated manner inthat database. Automated IVR voices may be detected either by building avoice model for each IVR voice, or detecting certain known prompts, orby detecting the parametric consistency of an IVR voice, or the like.

In a preferred embodiment of the present invention, when use of averbally programmed re-dialer is detected, the present invention has thecapability to blank the audio to the inmate's side of the line whileautomatically programming the verbally programmed re-dialer to dial adifferent number than the one spoken by the inmate. The inmateexperiences an unusual delay but does not know what has happened. Onepreferred embodiment then allows an investigator to converse with theinmate through voice-disguising software which may be tuned to mimic thevoice characteristics of the person the inmate was attempting to dial.

In another aspect, the present invention may be configured to repeatedlysteganographically embed user-identifying information in the audiostream of any recording or live call being streamed to any user. Thisfeature is intended to help institutions track down users who releasephone call audio to persons not authorized to have such audio.

In another aspect, the present invention supports creation ofpassword-protected audio files and encrypted audio files when audio isto be released outside of an institution within which it was recorded.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a screen shot of a graphical user interface according to thepresent invention, before any data is selected to be displayed.

FIG. 2 is a screen shot of a graphical user interface according to thepresent invention, showing an example menu of possible ways of selectinga span of time over which to analyze phone calls.

FIG. 3 is a screen shot of a graphical user interface according to thepresent invention, showing a pop-up calendar which may be used as an aidto selecting a range of time over which to analyze calls.

FIG. 4 is a screen shot of a graphical user interface according to thepresent invention, showing how the position of a second menu within theGUI may change depending on what is selected on a first menu.

FIG. 5 is a screen shot of a graphical user interface according to thepresent invention, showing how a second menu is used to select the typeof data to be displayed from the selected span of time.

FIG. 6 is a screen shot of a graphical user interface according to thepresent invention, a table which was dynamically populated based oncriteria specified by menu-selected time range and menu-selected type ofdata.

FIG. 7 is a screen shot of a graphical user interface according to thepresent invention, showing a pop-over graphical menu allowing aninvestigator to mark a call after having listened to all or part of thatcall.

FIG. 8 is a screen shot of a graphical user interface according to thepresent invention, showing a pop-over note pad which allowsinvestigators to enter notes concerning a given call.

FIG. 9 is a screen shot of a graphical user interface according to thepresent invention, showing how a second table is dynamically populatedwith data about likely imposters when a given suspect call is selected.

FIG. 10 is a screen shot of a graphical user interface according to thepresent invention, showing the suspected imposter table replaced by atable of notes on a given suspected imposter.

FIG. 11 is a screen shot of a graphical user interface according to thepresent invention, showing three dynamically generated tables(suspicious call report table, table of likely imposters concerning aselected call, and table of calls made by a selected inmate).

FIG. 12 is a screen shot of a graphical user interface according to thepresent invention, showing three dynamically generated tables, where theelevator bars to the side of the first two tables have been slid downsome.

FIG. 13 depicts speech from two participants in a conversation displayedseparately within a graphical user interface according to the presentinvention.

FIG. 14 depicts a speech segment from one participant in a conversationhaving been manually selected within a graphical user interfaceaccording to the present invention.

FIG. 15 is a graphical user interface for reviewing recordedconversations according to one aspect of the present invention.

DETAILED DESCRIPTIONS OF SOME PREFERRED EMBODIMENTS

Within this document, the terms “voice print”, “voice signature”, “voiceprint data”, “voice signature data”, and “voice model” may all be usedinterchangeably to refer to data derived from processing speech of agiven person, where the derived data may be considered indicative ofcharacteristics of the vocal tract of the person speaking. The terms“speaker identification” and “voice identification” may be usedinterchangeably in this document to refer to the process of identifyingwhich person out of a number of people a particular speech segment comesfrom. The terms “voice verification” and “speaker verification” are usedinterchangeably in this document to refer to the process of processing aspeech segment and determining the likelihood that that speech segmentwas spoken by a particular person. The terms “voice recognition” and“speaker recognition” may be used interchangeably within this documentto refer to either voice identification or voice verification.

Within this document, speech mannerisms will be deemed to include use ofparticular combinations of words (such as colloquialisms), frequency ofuse of various words (such as swear words, both as modifiers for otherwords, and not as modifiers), frequency and context of use of words andsounds some people use as conversational filler (such as the sound“aah”, or the word “like”), phrases which may be habitually used tostart speech segments (such as the phrase “OK”, or the word “so”),regional accents, elongation of pronunciation of words (for instancewhere the elongation may indicate a state of mind such ascontemplation), etc.

Within this document, the term “enrolled participant” will refer to aparticipant who's voice or other biometric identifying data (such asfingerprint data, retina scan data, or photographic data) is enrolled inthe system, who has been recognized by the system as enrolled. Inapplications of the present invention concerned with identifyingconversation participants or controlling in some way who participates ina conversation, the terms “allowed” and “authorized” when used todescribe a conversation participant will refer either to a conversationparticipant who's voice or other biometric identifying data isrecognized as enrolled in the system (and who is authorized toparticipate in a given portion of a conversation without an automatedaction being taken), or a conversation participant who is identified aspart of a class of participants (such as women, or men, or persons who'svoice or other biometric identifying data is not recognized), and who,based on his or her individual identity or class identity, is permittedto participate in a given portion of a conversation without an automatedaction being taken (such as an alert being generated and/or an automatedmessage being played and/or a call being disconnected).

Within this document, the terms “unauthorized” and “unallowed”, whenused to describe a conversation participant, will refer to persons whohave not been identified as “allowed” or “authorized”. Within thisdocument, the term “disallowed”, when used to describe a conversationparticipant, will refer either to a conversation participant who's voiceor other biometric identifying data is recognized as enrolled in thesystem, or a conversation participant who is identified as part of aclass of participants (such as women, or men, or persons who's voice orother biometric identifying data is not recognized), and who, based onhis or her individual identity or class identity, is prohibited fromparticipating in a given portion of a conversation.

Within this document, the term “inter-prosodic” shall refer to allnon-vocabulary-based characterization of the quality of conversationbetween two or more persons. For instance a conversation between twopeople might be characterized as confrontational, friendly, teaching,order giving/taking cooperative, etc. The term “trans-prosodic” withinthis document shall refer to non-vocabulary-based patterns of thequality of a given speaker's speech across a substantial portion of aconversation or lengthy speech segment (for instance, across a portionof a conversation or speech segment long enough for an emotional patternto be distinguishable).

Within this document, the term “key-phrase” shall be used to denote anyword, phrase, or phonetic utterance which one might wish to detect orsearch for in a conversation. Within this document, the term “callcontrol official” shall be used to refer to any person who has theauthority to interrupt a phone call which is in progress. Suchinterruption could be by joining the call, temporarily disconnecting oneor more parties from the call, disconnecting the entire call, ordisabling one or more parties ability to be heard on the call.

In a preferred embodiment, when a user of the graphical user interfaceof the present invention first sits down to use the system, he sees ascreen such as the one shown in FIG. 1, which may usefully be thought ofas consisting of five regions as follows: Call Report Selection Area 101allows the user to select a span of time and a type of calls to displayover that span of time. Call Report Area 102 contains a note that thisarea will be used to display the call report on the data selected inCall Report Selection Area 101. Call Analyzer Area 103 contains a notethat this area will be used to display a table of information relevantto whatever call is selected on the call report table which will begenerated in Call Report Area 102. Call Records Selection Area 104allows the user to select a set of call records via menu criteria suchas phone number, inmate ID, etc. Call Records Area 105 contains a noteexplaining that a table of call records will be generated in that areagiven criteria specified in Call Records Selection Area 104.

FIG. 2 shows how time-span menu 200 expands (showing selectable options)when time-span menu drop-down arrow 201 is clicked. In a preferredembodiment, time-span menu 200 contains selections “Yesterday andToday”, “Today”, “Last X days”, “Since . . . ”, “Past Week”, and “PastMonth”. Depending on which selection is chosen, dynamically generatedadditional time-span data box 202 will appear. For instance, if “Today”is selected, box 202 will not appear, whereas if “Last X days” isselected, data box 202 will appear. If “Since . . . ” is selected,dynamically generated pop-over calendar 300 will appear, as shown inFIG. 3. Pop-over calendar 300 may also include time field 301, intowhich a user may type a time of day, if desired. In an alternateembodiment, time span menu 202 may include a “Between” selection, and atime range may be selected in pop-over calendar 300.

Depending on what menu selection is selected from time-span menu 200,time-span data box 202 may change size and the position of other objectswithin the display area may change to accommodate this change. Forinstance, FIG. 4 shows call-type menu 400 changes position relative toFIG. 1 or 2 when “Since . . . ” is selected on time-span menu 200,making time-span data box 202 increase in size to accommodate a date ordate range.

FIG. 5 shows the selections available on call-type menu 500, which dropsdown when call-type drop-down arrow 501 is clicked. In a preferredembodiment, selections on call-type menu 500 include “Suspicious Calls”,“Unrated Suspicious Calls”, “Officer Rated Calls”, “A Specific CSN”,“Calls Using This Inmate ID # . . . ”, and “Calls to This Phone # . . .”. If any of the selections containing “ . . . ” are selected, then anadditional call-type data box appears, allowing the user to typeadditional call type data (such as a phone number, Call Sequence Number(CSN), or inmate ID number).

Once a user has entered time-span and call-type data, Recorded CallSearch Button 600 can be clicked to dynamically generate report table601, as shown in FIG. 6. Column 1 (the left-most column) in report table600 is both numerically coded and color coded to indicate level ofseverity. For instance, in the Suspicious Call Report shown in FIG. 6,the table is sorted (from top to bottom) in order of how likely it isthat the voice on the call is the voice of an imposter. The color red isused to represent a highly suspicious call, while the color green isused to represent a very-low-suspicion call.

The second column in table 601 contains the call sequence number (CSN).A unique CSN is assigned to each call made. The second column also hasspace for a note-denoting icon such as icon 602, which will be presentif an official has entered any notes concerning the call. The thirdcolumn in table 601 contains the date of the call. The fourth column intable 601 contains the ID number used to place the call. If the call issuspected to be made by an imposter, it is suspected that this numberdoes not belong to the person who's voice is actually heard on the call.The fifth column of table 601 contains the name that goes with the IDnumber is column 4, plus a small icon of a man with sun glasses if ithas been determined that the call was actually placed by someone else.

Coolum 6 of table 601 contains icons indicating the rating status ofeach call. In a preferred embodiment, if a call has not been rated by anofficial, a question mark icon will be present. When an official listensto a call, he may right-click on the icon in column 6 of table 601, andpop-over window 700 will appear (as shown in FIG. 7), allowing the userto select a rating Icon of his choice. In a preferred embodiment, asun-glasses icon represents the rating “imposter”, a halo iconrepresents the rating “innocent” (not imposter), and a balance iconrepresents the rating “unsure” (of whether the person actually speakingon the call is the person who's ID number was used to place the call).If a user wishes to attach a note to a call record displayed in table602, he may do so by right-clicking in column 2 of the call record, anda note window 800 will appear as a pop-over, allowing the user to type anote. In an alternate embodiment, notes may also be entered as voiceannotation.

In a preferred embodiment, when a row of table 601 is selected (forinstance, by left-clicking anywhere in the row, then call analyzer table900 is dynamically generated, and the row of table that generated table900 is shaded as row 901 is shaded in FIG. 9. In a preferred embodiment,each row of table 900 represents a possible imposter, with the tablesorted such that the most likely imposter is at the top, andsuccessively less likely imposters appear on the rows below. Column 1(the left-most column) of table 900 contains the possible imposter'sname. Column 2 contains the possible imposter's ID number. Column 3 is acolor-coded and numerically coded column containing the automaticallygenerated Suspicious Call Finder (SCF) rating. In a preferredembodiment, the SCF rating is generated based on two factors. The firstfactor is the similarity of the possible imposter's voice to the voiceheard on the call. The second factor is the frequency with which thepossible imposter has been known to dial the number that was dialed(either under his own ID or on calls where he has previously beenverified to be identifying himself as someone else).

Column 4 of table 900 contains the number of calls the suspectedimposter has made to the dialed phone number in the past. Column 5 oftable 900 contains the amount of money the suspected imposter has spentcalling the dialed number in the past. Column 5 contains the percentageof all calls made to the dialed number which were made by the suspectedimposter. Column 6 contains an icon which indicates whether thesuspected imposter's voice has been verified by an official to be on thecall. Note that it is possible for more than one imposter to participatein a call, so some embodiments may allow more than one “verified” iconin column 6. In a preferred embodiment, the checkmark icon, the questionmark icon, and the halo icon in column 6 represent “verified imposter”,“unknown”, and “innocent” respectively.

In a preferred embodiment, the inmates whose voices and call historieswere analyzed to generate possible imposter table 900 are chosenautomatically based on system records indicating who has access to thephone from which the suspicious call was placed. Sometimes records maynot be updated quickly when inmates are moved, so button 901 is providedso that officials can manually add inmates to the list of possibleimposters. When inmates are manually added, on-the-fly computation willbe done to rate the added inmate's voice and call history to rank thatinmate among the possible imposters. Notes previously entered for thecall selected in table 601 may be viewed by clicking button 902.Clicking button 902 replaces possible imposter table 900 with notestable 1000 (shown in FIG. 10). The user may switch back to viewingpossible imposter table 900 by clicking button 1001.

In a preferred embodiment, a third dynamically generated table 1100 maybe generated by selecting a menu item from Call Records Search Menu1101, and filling in associated Call Record Search Data Box 1102. Forinstance, in FIG. 11, call records table 1100 represents all the callsmade by inmate number 165388, and in FIG. 12, call records table 1200represents all the calls made to the phone number filled in in box 1102.In a preferred embodiment of the present invention, when dynamicallygenerated tables (such as 601, 900, or 1200) do not fit in theirallotted space either horizontally or vertically or both, verticaland/or horizontal elevators such as 1201, 1202, 1203 allow verticaland/or horizontal scrolling of the table which does not fit in itsallotted space. Likewise, the entire display window may be scrolled byweb browser elevator 1204. In a preferred embodiment, the graphical userinterface of the present invention is implemented using asynchronousJava and XML (AJaX) programming standards which have evolved forwebsites and the like.

In a preferred embodiment for use in correctional institutions, agraphical user interface such as shown in FIG. 13 may display inmateconversation waveform 1300 and called-party conversation waveform 1301separately.

In a preferred embodiment for use in correctional institutions, agraphical user interface such as shown in FIG. 14 allows call reviewpersonnel to highlight a portion of a conversation participants speech(such as inmate speech segment 1400), and build a speech model,voiceprint, voice signature of the like from that segment by clickingVoice Modeling button 1401. In a preferred embodiment, this functionleads to a dialog box which allows the user to build a voice model orvoiceprint, and store any voice model or voiceprint produced for furtheruse. A preferred embodiment of the present invention allows automatedsearching of past conversations to detect voices of conversationparticipants whose voice characteristics fit a given voice model orvoiceprint.

A preferred embodiment of the present invention for use in correctionalinstitutions allows officials to automatically flag future calls onwhich a voice matching a given voiceprint or fitting a given voice modelis detected.

Thus a preferred embodiment of the present invention allows correctionalofficers to quickly specify a segment of speech within a graphicalinterface, and search a set of past phone calls and/or flag future phonecalls when the voice of the person speaking in the specified speechsegment is detected on such calls. This allows easy searching forpatterns of identity theft and the like, once an identity theftperpetrator or suspect has been identified by voice segment.

A preferred embodiment of the present invention for use in financialinstitutions similarly facilitates the building of a voice model orvoiceprint from a speech segment of a call on which a credit card fraudperpetrator or suspect's voice segment has been identified, andsimilarly allows the automated searching of past recorded conversationsfor occurrences of that person's voice. Such preferred embodiment alsoallows automated real-time flagging of calls where such perpetrator'svoice is detected, allowing improved fraud prevention, and facilitatingautomated call tracing to aid law enforcement officials in apprehendingcredit card fraud suspects.

For example, if a customer named Fred had a credit card, and a creditcard fraud perpetrator were to obtain Fred's personal identifyinginformation (such as social security number and mother's maiden name),call Fred's credit card company, pretend to be Fred, claim to havemoved, and have a new credit card sent to a different address, once thefraud was detected, the credit card company could (using the presentinvention) find the archived phone conversation, make a model of theperpetrator's voice, automatically detect instances of the perpetrator'svoice on the phone in the future, and automatically block future fraudattempts of the perpetrator, while simultaneously in real time alertinglaw enforcement officials of the phone number the perpetrator calls infrom.

In a preferred embodiment, call review officials can use a graphicalinterface such as shown in FIG. 14 to mark a speech segment which isexemplary of a given emotional state of given speaker, and model of thatemotion in that person's voice can be stored. Subsequently, pastconversations in which that speaker participated can be searched forinstances of similar emotional states. Likewise, future conversationsmay be monitored in real time to detect the occurrence of a similaremotional state, and such detection can be configured to flag suchconversations for later review, or alert personnel in real time whensuch conversations are taking place.

In a preferred embodiment, call review officials can use a graphicalinterface such as shown in FIG. 14 to mark a speech segment which isexemplary of a given speech mannerism or phonetic pattern, and store amodel of that speech mannerism or phonetic pattern. Subsequently, pastconversations can be searched for instances of a similar speechmannerism or phonetic pattern. Likewise, future conversations may bemonitored in real time to detect the occurrence of a similar speechmannerism or phonetic pattern, and such detection can be configured toflag such conversations for later review, or alert personnel in realtime when such conversations are taking place.

In a preferred embodiment of the present invention for use in financialinstitutions, certain conversation participants are automaticallydetermined to be “genuine” (either in real time or by subsequentevidence), and speech models or voiceprints are automatically built andsaved from such verified genuine recordings to aid in fraud preventionon future calls. For example, if within a given conversation a customerarranges to make a particular payment, and such arranged payment issubsequently made, then, once that payment has been made, the voice inthat original recorded conversation can be assumed to be the voice ofthe customer.

Key points for Financial Services fraudulent transaction detector.

-   -   I. Classify Customer Service to Customer phone conversations as        potential or non potential fraud transactions.        -   A. Potential Fraudulent transactions (examples)            -   i. Change of Address            -   ii. New credit card request            -   iii. Change in password            -   iv. Credit limit increase requests        -   B. No Fraud Potential Transactions (examples)            -   i. Clarification of a charge            -   ii. Arrangements for direct payment    -   II. Build voice models by credit card number or corresponding        customer ID        -   A. Build Voice models for No Fraud Potential Transactions—            -   i. If significantly different than previous voice model,                build a new model            -   ii. If similar to previous voice model, add it to                previous voice model and make a new model        -   B. Build Voice models for Fraudulent transactions            -   i. When a confirmed impostor is detected, build a voice                model for the transaction and add the voice model to the                impostor models    -   III. Fraud detection        -   A. When calls come into the Call Center            -   i. Check transaction type            -   ii. If Potential Fraudulent transaction use JLG                Technologies software to look for impostor voice model                matches            -   iii. Provide real time alerts on high potential loss                calls            -   iv. Use lie detection software on high potential fraud                calls        -   B. Post Call Processing            -   i. If a call is considered fraudulent after the fact,                add the perpetrators voice to the impostor group and use                this as part of the fraudulent voice detection group    -   IV. Reporting        -   A. The system generates reports to provide potential            fraudulent activity            -   i. Low scoring voice model customer calls that have high                scoring impostor scores            -   ii. Use lie detection on high potential fraud calls

In a preferred embodiment, the voice model used to ongoingly verify theidentity of a conversation participant is automatically updated due tocircumstances detected in the conversation. For instance, if it isdetermined that the conversation participant Fred has switched fromspeaking to an adult to speaking to a child, a voice model previouslyderived during a speech segment when Fred was speaking to a child wouldbe substituted for the model used when Fred is speaking to an adult. Ina preferred embodiment, the boundaries of the speech segments when Fredis speaking to an adult and when Fred is speaking to a child would bedetermined by speech-to-text conversion, and detection of phrases spokenby Fred such as “put <name> on the phone”, or “let me speak to <name>”,or phrases spoken by another conversation participant such as “I'll put<name> on the phone”, or “here she is”, or “here's <name>”. In apreferred embodiment, a second detected circumstance which would cause aswitch in the voice model used to ongoingly identify Fred would be thedetection of a new conversation participant's voice just before thespeech segment from Fred for which the second voice model is used.

In the above example, in a preferred embodiment, a third detectedcircumstance that will cause the system to switch to using a differentvoice model for ongoing verification of Fred's identity would be thedetection of speech from Fred's side of the line (determined, forinstance, by telephonic directional separation in an institutional phonecall) which is no longer easily verifiable as coming from Fred using thevoice model used to verify Fred's previous conversation segment. In apreferred embodiment, when the system switches to using a differentvoice model for Fred, the system will keep using that voice model untilthat voice model is no longer a good match, or until circumstances aredetected that indicate the system should switch to a different model. Asabove, such circumstances may include detected words spoken by Fred oranother conversation participant or both, and such circumstances mayinclude a change in the detected voice of the person with whom Fred isconversing.

In a preferred embodiment for use in applications where it is desirableto perform ongoing identity verification via voice, when the calculatedprobability that an imposter is speaking rises above a certain level (orthe calculated probability that the correct speaker is speaking fallsbelow a certain level), the system performs an automated search for abest match of the speaker's voice with voice models of suspectedimposters. In a preferred embodiment, this imposter search is done in anefficient manner. In one preferred embodiment, voices of persons whocould be imposters are divided into a plurality of groupings based oncertain measured characteristics of the voice, and only voice modelswithin such grouping as the voice being checked would fall are checkedfor a match to the possible imposter speaking.

In the literature, persons who's voices get classified into one of aplurality of classifications are sometimes referred to as “cohortspeakers”. The concept of cohort speakers can be thought of as a way tosave time searching for speaker's identity, by comparing a small regionof one speaker's speech-feature space with like region of anotherspeaker's speech-feature space, rather than doing an exhaustivecomparison of the entire feature space. This is analogous of comparing aportion of a finger print, and only going further in the comparison ifthat portion appears to be a match.

In a preferred embodiment, another way in which the search is done in anefficient manner is to limit the set of potential imposters to the setof persons known to have access to the phone on which the potentialimposter is speaking, at the time the potential imposter is speaking.This limitation feature would, for instance, be useful in limiting theset of potential imposters who might impersonate a given prison inmate,but it would not be as useful in applications such as ongoing voiceverification of customers during telephone banking and other financialtransactions which might be carried out over the phone. In suchfinancial applications, potential imposter voice models to be searchedcould, for instance, start with voice models derived from voicespreviously recorded in transactions which were later determined to befraudulent.

In a preferred embodiment for use in prisons, rather than simplybeginning with voice verification (based on an assumed identity of aninmate from PIN provided or the like), the system starts by performingvoice identification, based on a maximum-likelihood search within a setof voice models believed to be representative of all possible personswho could be speaking on a given telephone. Once the most likelyidentity is derived, that identity is compared to the claimed identityof the inmate speaking. If the claimed identity and themaximum-likelihood-search-derived identity are identical, then thesystem switches into ongoing voice verification mode.

In a preferred embodiment of the present invention for use inapplications where the set of possible conversation participants isknown and closed (for instance in a correctional institution, where theset of all possible persons who might use a given telephone is known),the present invention first identifies which voice model or voiceprintmost closely matches the voice characteristics of a conversationparticipant being identified. In such an embodiment, the presentinvention then switches to ongoing-voice-verification mode. In such apreferred embodiment, the threshold used by the present invention todecide whether the identity of the conversation participant beingmonitored has changed is chosen based upon the difference between howfar off the parameters of the voice of the person speaking are from theparameters of the voice model the speaker has been identified to beassociated with compared to how far off the parameters of the voice ofthe person speaking are from the parameters of the voice model of thenext most likely person who the speaker might have been identified as.

An example is provided here for added clarity. Suppose the set ofpossible conversation participants within a given prison cellblockconsists of Fred, Bob, Joe, Ted, and Al. One of these men gets on thephone and begins speaking. The present invention compares the voiceparameters of the man who is speaking with the previously stored voiceparameters of Fred Bob Joe Ted and Al. The closest match is found to beAl. The next closest match is found to be Ted. After the conversationparticipant has been identified as Al, the system switches into voiceverification mode. If the voice parameters of the person speakingclosely match the stored voice parameters of Al, and the voiceparameters of the person speaking are very different from the storedvoice parameters of Ted, a “large” tolerance will be allowed in theongoing identity verification of Al as the conversation progresses. Onthe other hand if the voice parameters of the person speaking closelymatch the stored voice parameters of Al, and the voice parameters of theperson speaking are also fairly close to the stored voice parameters ofTed (though not as close as they are to the voice parameters of Al), a“small” tolerance will be allowed in the ongoing identity verificationof Al is the conversation progresses. Also, if the voice parameters ofthe person speaking are not very close to the stored voice parameters ofAl but they're a bit closer to the stored voice parameters of Al thanthey are to the stored voice parameters of Ted, then again a “small”tolerance will be allowed in the ongoing voice verification of Al.

Thus, in applications where there is little tolerance for errors invoice identification and/or verification, the present invention allowslarger variations in a given person's voice before identifying suchperson as an imposter in closed-population situations where no otherpersons have similar voices than the present invention allows beforeidentifying such person as an imposter in closed-population situationswhere one or more other persons have voices similar to the voice of theperson speaking Thus, the present invention may be thought of in suchapplications as having an “imposter detection” threshold which isrepresentative of the amount by which the voice parameters of a givenconversation participant are allowed to very during a conversationbefore that participant is considered an imposter, and the imposterdetection threshold for different conversation participants within agiven closed set of possible conversation participants will differ fromone conversation participant to another, depending on whether there areother persons within the possible set of conversation participants whosevoices are similar to the conversation participant whose voice is beingidentified and/or ongoingly verified within a conversation.

Within this document, the variation tolerance allowed for the measuredparameters of a given speaker's voice during ongoing voice verificationwithin a given conversation will be referred to as that speaker's voiceidentity tolerance within that conversation. Voice identity tolerancemay be defined one-dimensionally or multi-dimensionally, depending onthe nature of the voice parameter measurements made in a givenembodiment of the present invention.

In a preferred embodiment of the present invention incorporating initialvoice identification followed by subsequent ongoing voice verification,should the ongoingly monitored parameters of the speaker's voice very bymore than the allowed voice identity tolerance (or imposter detectiontolerance), the system switches back from voice verification mode intovoice identification mode, and once again the most likely identity ofthe speaker is chosen from within a known closed possible set ofspeakers, based upon a maximum likelihood search.

In a preferred embodiment of the present invention employing acombination of voice identification and voice verification techniques,statistics indicative of the certainty to which a given conversationparticipant has been identified in a given conversation are stored alongwith the voice data and/or text data derived from that conversationthrough speech-to-text conversion.

In a preferred embodiment, more than one voice model or voiceprint maybe stored for a given conversation participant. For instance, if it isfound that the voice parameters of a given conversation participant aredistinctly and predictably different when he or she speaks to a child onthe phone versus when he or she speaks to an adult on the phone it maybe advantageous to store two separate voice models or voiceprints, thenit would be to try and average the voice parameters of that conversationparticipant between those two different modes of speaking.

In a preferred embodiment of the present invention employing ongoingvoice identity verification within a known close population, andemploying voice characterization of an n-dimensional nature, then-dimensional distance from the voice model of the assumed “correct”conversation participant to the voice model of the “most likelyimposter” within the closed population is determined, and ongoing voiceverification is done in such a way that if the ongoingly measured voiceparameters of the person speaking drift away from their ideal positionin voice parameter space by more than half the distance to the closestpossible imposter than a “likely imposter” alarm condition is generated.

It can be seen that within a multidimensional model space, while“maximum allowed drift distance” condition determination is optimal ifthe drift in parameters of the speaker's voice parameters go in theexact direction of the n-dimensional position of the most likelyimposter's voice, this alarm condition's tolerance may be unnecessarilytight if the drift of the ongoingly measured voice parameters of thespeaker within the n-dimensional space are in a direction other than thedirection toward the most likely imposter, and the next most likelyimposter is significantly further away in the n-dimensional space thanthe first most likely imposter is. Plus, in a preferred embodiment theallowed drift distance (within n-dimensional voice model space) of theongoingly measured voice parameters of the conversation participantdiffers depending on the direction (within n-dimensional parameterspace) of the drift detected.

During the process of ongoing voice identity verification, the shorterthe speech segment used to derive voice model parameters is, the lessaccurate the derived voice model parameters can be assumed to be.However, if the length of a speech segment used to verify identity of aconversation participant is inadvertently made so long that part of itmay contain speech from more than one conversation participant, then theidentity verification accuracy available from the speech segment isactually less than the identity verification accuracy available from ashorter speech segment which only contains speech from one conversationparticipant.

In a preferred embodiment of the present invention for use incorrectional institutions and the like, the length of speech segmentsused for ongoing identity verification are chosen based on the boundaryconditions of those speech segments. Since it is much more likely thatan imposter will enter a conversation immediately following a pause inspeech on the side of the line on which the imposter enters theconversation, normally such pauses are used to determine the boundariesof speech segments used in ongoing identity verification. However, it isalso possible that inmates in a correctional institution will learn ofsuch an algorithm and will attempt to have an imposter join aconversation by overlapping the imposter's speech with the speech of the“correctly” verified conversation participant. A preferred embodiment ofthe present invention therefore employs a second speech segment boundarydetermining mechanism which operates by detecting the simultaneouspresence of two voices. Such a detection may be done, for instance,through phase locked frequency domain measurements, or time domainmeasurements which detect the presence of the periodic pulse trains ofmore than one set of vocal chords at the same time.

In addition to the use of voiceprints and voice models (such as Gaussianmixture models) known in the art, in a preferred embodiment, the presentinvention additionally utilizes detection of and classification ofspeech mannerisms of conversation participants as part of the evidenceused to ongoingly verify the identity of conversation participants. In apreferred embodiment, measured speech mannerisms include statisticalprobabilities of use of words accents timing and the like derived fromprevious known samples of a given conversation participant's speech.

Real-world applications often make computational efficiency an importantbenchmark of system performance. Within the present invention one of themethods used to increase computational efficiency without sacrificingtoo much in the way of false “imposter” detection, is not to analyze allof the speech within a given speech segment. In a preferred embodimentspeech segments which are considered to be “long”, and which areconsidered to be highly unlikely to contain speech from more than oneindividual only need to have a fraction of their speech content analyzedfor identity verification purposes. As long as “enough” of the speechwithin the segment is analyzed, the certainty to which identityverification is carried out can be fairly high. One way in which thepresent invention is able to select a subset of speech to be analyzed,is to analyze every Nth 10 ms of speech, where the number N is chosen tobe small enough to allow reliable voice Identity verification. It shouldbe noted that this style of data reduction may not be usable insituations where speech mannerisms including temporal patterns andspeech are being analyzed as part of the identity verification process.

When a speech feature space is mathematically defined such that acertain area of that speech feature space is occupied only by featuresof the speech of a known subset of possible speakers, the mapping ofsuch known subsets of possible speakers to areas of such a speechfeature space is known as a “codebook”. It is mathematically possible todefine a codebook such that the known subsets of possible speakersreferred to by the codebook are each individual speakers. In a casewhere the codebook is defined such that the known subsets of possiblespeakers referred to by the codebook each contain a group of possiblespeakers, the individuals in such a group of possible speakers arereferred to as “cohort speakers”.

In a preferred embodiment, digital signal processing involved inderivation of voice prints, voice models, phonetic sequences, and textfrom audio is carried out on DSP processors on boards commerciallyavailable graphics cards such as those used for high-end computer games.In a preferred embodiment, one or more such graphics boards is installedin a personal computer (PC) to implement the computational hardware ofthe present invention. Carrying out digital signal processing on suchcards facilitates far more economical implementation of the features ofthe present invention than implementing all the computation of thepresent invention on typical personal computer CPUs.

A call monitor/review interface according to a preferred embodiment ofthe present invention is shown in FIG. 15. Graphical face symbol/namepairs such as 1506 are added to the display as they are detected in alive monitoring application, or all appear when a call to be reviewed isaccessed. In a preferred embodiment, during call monitoring using asimple telephone interface, pressing a particular DTMF digit on themonitoring phone will cause the present invention to enunciateidentifying information about the individual speaking.

Within this document, means for prompting shall be construed to includerecorded voice prompt played under computer control, synthetic voiceprompts played under computer control, text prompting on a visualcomputer-controlled display, prompting by displaying detected conditioninformation stored in a database in response to a database query, andany other means for prompting known in the art. Within this document,means for deriving a voice print or voice model shall include means suchas referenced directly or indirectly in U.S. Pat. No. 7,379,868, and anyspeaker identification and voice identification publications known inthe art.

Within this document, means for recording voice signals from telephoneconversations shall be construed to include means disclosed in documentssuch as U.S. Pat. Nos. 7,844,252 and 7,889,847 (which are hereinincluded by reference), and hardware such as available from Adtrans forconverting analog telephone signals to digital form, and compressingdigital speech streams by known compression algorithms such as G729, andhardware which implements a SIP stack and communicates compressed oruncompressed digital audio in packet form over the Internet.

Within this document, the term loudspeaker shall be construed to includeear-mounted audio speakers such as headphones, earbuds and earphones aswell as conventional free-standing loudspeakers.

Within this document, means for monitoring a telephone line for asuspected 3-way call event includes an analog-to-digital converter andmay include one or more means known in the art for detecting a 3-waycall event, such as silence detection means, click and pop detectionmeans, keyword detection means, special information tone detectionmeans, means for detecting a change in voice characteristics, lineimpedance change detection means, volume level change detection means,means for distinguishing yelling, or any combination of these means.

Within this document, each of the following are considered to besuspicious call criteria:

-   -   identity scamming detection    -   3-way calls    -   spoken phone number detected    -   added called party    -   high interest group inmate    -   high interest group called party    -   newly identified link    -   matches historical prior confirmed suspicious calling pattern    -   new called party voice seen on call.    -   previously identified high interest called party voice detected    -   whispering detected    -   prior inmate voice detected

For the purposes of this patent application, the above list shallconstitute the complete list of suspicious call criteria.

Within this document, the term “voice characteristic data” shall be usedto refer to voice model data or voice print data derived from a sampleof spoken utterances from an individual. Within this document, thephrase “alerting a call control official” shall be deemed to construeeither: placing a phone call to a call control official and playing anaudio message, or sounding an audio alarm that can be heard by a callcontrol official, or displaying a visual indicator that can be seen by acall control official, or displaying detected condition informationstored in a database in response to a database query, or actuating avibrating alarm that can be felt by a call control official, or somecombination of these. In this document, the term “call-restrictingaction” shall be construed to mean either alerting a call-controlofficial as to the nature of the call that may merit being cut off, orsimply automatically cutting off the call.

Within this document, the term “directional microphone” shall includenot only any microphone which is inherently acoustically directional,but also any array of microphones whose signals are processed in eitherthe analog or digital domain to produce a single output signal such thatthe array behaves like a directional microphone. If all the signals froman array of microphones are recorded in a time-synchronized manner, therecorded signals may be post-processed in different ways after the factto selectively listen in different directions.

The foregoing discussion should be understood as illustrative and shouldnot be considered to be limiting in any sense. While this invention hasbeen particularly shown and described with references to preferredembodiments thereof, it will be understood by those skilled in the artthat various changes in form and details may be made therein withoutdeparting from the spirit and scope of the invention as defined by theclaims.

The invention claimed is:
 1. A prison phone monitoring systemcomprising: an audio database containing digitized audio signals storinga plurality of voice models and a plurality of recorded telephone calls;a call monitoring system for interfacing with a call network formonitoring a first telephone call, wherein the call monitoring system isconfigured to: detect a 3-way call event, wherein the 3-way call eventindicates a new voice joining the monitored first telephone call; searchthe audio database for a voice model matching the new voice; determine aphone number for an individual associated with the matching voice model;and utilizing a call network query to determine a busy status of thephone number during the first telephone call.
 2. The prison phonemonitoring system of claim 1, wherein the monitoring of the firsttelephone call further comprises recording the first telephone call andsaving the recorded first telephone call to the audio database.
 3. Theprison phone monitoring system of claim 1, wherein the call monitoringsystem is further configured to: create a new voice model correspondingto the new voice, if no voice model matching the new voice is found inthe audio database.
 4. The prison phone monitoring system of claim 1,wherein the determined phone number is associated with the matchingvoice model based on prior monitored communications by the individualusing the phone number.
 5. The prison phone monitoring system of claim4, wherein the busy status of the phone number is determined using thecall network query, wherein the call network query cannot be detected bythe individual.
 6. The prison phone monitoring system of claim 1,wherein the busy status is ascertained using automated dialing means. 7.The prison phone monitoring system of claim 1, wherein the call networkquery comprises a SIP (Session Initialization Protocol) request issuedto the determined phone number.
 8. A method for monitoringcommunications within a prison phone system comprising: storingdigitized audio signals corresponding to a plurality of voice models anda plurality of recorded telephone calls in an audio database; monitoringa first telephone call using a call monitoring system configured tointerface with a call network; detecting a 3-way call event in themonitored first telephone call, wherein the 3-way call event indicates anew voice joining the monitored first telephone call; searching theaudio database for a voice model matching the new voice; determining aphone number for an individual associated with the matching voice model;and utilizing a call network query to determine a busy status of thephone number during the first telephone call.
 9. The method of claim 8,wherein the monitoring of the first telephone call further comprisesrecording the first telephone call and saving the recorded firsttelephone call to the audio database.
 10. The method of claim 8, furthercomprising: creating a new voice model corresponding to the new voice,if no voice model is found in the audio database that matches the newvoice.
 11. The method of claim 8, wherein the determined phone number isassociated with the matching voice model based on prior monitoredcommunications by the individual using the phone number.
 12. The methodof claim 11, wherein the busy status of the phone number is determinedusing the call network query, wherein the call network query cannot bedetected by the individual.
 13. The method of claim 12, wherein the busystatus is ascertained using automated dialing means.
 14. The method ofclaim 12, wherein the telephone network query comprises a SIP (SessionInitialization Protocol) request issued to the determined phone number.